Issue Brief 


August 2021 
Understanding the Test Preparation 
Practices of Underserved Learners 


RAEAL MOORE & SWEET Z. SAN PEDRO 


ACT.org ACT 


ACT Research & Policy | Issue Brief | August 2021 


—_ 


Understanding the Test Preparation Practices 
of Underserved Learners 


Raeal Moore & Sweet Z. San Pedro 


Introduction 


The ACT® test is a pivotal step in the college-going process for students across the 
United States. Many students take college entrance exams, like the ACT, to highlight 
the knowledge they have gained in K-12 and their level of preparedness for college. 
Likewise, colleges use these exams in the admissions review process because they 
provide a common basis for comparing students across schools and states. The 2020 
graduating class, for example, had over 1.6 million students take the ACT and over 2.2 
million take the SAT. 


Given the importance of test scores in college admissions decisions and scholarships, 
a test preparation and tutoring industry has proliferated (Barnes Reports, 2017). In 
fact, in 2018, 64% of students who took the ACT indicated that they participated in at 
least one test preparation activity.' Test preparation can include activities such as 
using study guides, completing practice tests, hiring a tutor, or participating in a test 
preparation class. Test preparation solutions can be delivered in person, online, or 
using a hybrid format. Delivery can be synchronous or asynchronous. While test 
preparation activities vary, their unifying goals are to improve students’ knowledge of 
the content measured on the test and to provide students with test-taking strategies 
(Messick, 1982).2 


While research shows that, on average, engaging in any test preparation activities is 
better than engaging in none (Briggs, 2009; Moore, Sanchez, & San Pedro, 2018; 
Powers, 1993; Schiel & Valiga, 2014a, 2014b), the research also shows that certain test 
preparation features are more effective than others. For example, preparation is most 
effective in achieving higher test scores when test preparation activities align to 
rigorous high school coursework, since tests like the ACT and SAT measure students’ 
knowledge of such coursework. In addition, working with a private tutor or test 
preparation course is more effective at improving test scores than test preparation 
where students work on their own or in small group settings (Bloom, 1984; Ireson, 
2004; Moore et al., 2018). Limited research has shown that the amount of time 
allocated to online test preparation does not improve standardized test scores 
relative to the scores of those who engaged in other learning modes (Moore et al., 
2018). This aligns with research outside of test preparation, which has shown that 
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online learning is less effective than learning in person (Ahn & McEachin, 2017; 
Heppen et al., 2017). 


Unfortunately, research also shows that students from more affluent families enroll in 
test preparation activities at a higher rate than students who come from lower- 
income households (Buchmann, Condron, & Roscigno, 2010; Park & Becks, 2015). 
Moreover, the types of test preparation that students from higher-income households 
participate in, such as test preparation courses and private tutoring, are also the most 
expensive and most effective. If students from lower-income families do participate 
in test preparation activities, their engagement is more likely to be with less 
expensive resources like books or self-paced courses (Buchmann et al., 2010; Sanchez, 
2019). Furthermore, students who belong to households where standardized tests 
and test preparation options are discussed are also more likely to participate in test 
preparation. These types of conversations are more likely to occur when parents have 
gone through the college-going process (Institute of Higher Education Policy, 2012). 
Therefore, it is not surprising that first-generation college-going students are also less 
likely to participate in test preparation activities because of the absence of these 
critical conversations, financial constraints, or both. Participation in test preparation 
by race is more complicated. Black and Hispanic students, on average, tend to 
participate in more test preparation than their White counterparts, and this is 
especially true for the purchasing of private, and often expensive, test preparation 
activities for Black and Hispanic students who have a higher income and lower test 
scores (Buchmann et al., 2010; Park & Becks, 2015; Alon, 2010). 


Given these disparities, ACT wanted to better understand what test preparation 
activities and delivery formats students from different backgrounds used before 
taking a national standardized assessment and how effective these activities were in 
improving students’ ACT Composite scores. Therefore, we invited approximately 
45,000 high school students? who had registered for the February 2016 national 
administration of the ACT test to respond to an online survey. A total of 6,889* 
students did so. We investigated the ways in which students engaged in test 
preparation, whether underserved learners participated in different test preparation 
activities than their peers, and whether any of the test preparation features were 
related to how well students performed on the ACT.° 


Test Preparation Activities and Products 


Given the wide variety of test preparation activities and products available to 
students, we asked our participants to indicate which activities they used to prepare 
for the February 2016 national test administration, and then we categorized their 
responses into the following groups: pace-defined, modality-defined, and product- 
defined.°® 


ACT Research & Policy | Issue Brief | August 2021 2 


Pace-defined test preparation activities were defined based on who was directing 
the student's learning. These included instructor-led activities and self-paced 
activities. 


Modality-defined test preparation activities were defined based on their mode of 


delivery. These included online, not at all online (identified as “non-online”), and a mix 
of the two. 


Test preparation activities were classified (based on a single survey question) in terms 
of both their pacing and modality. Pace-defined and modality-defined test 
preparation activities were compared to the preparation activity of using only 
practice tests (which, by definition, means that the student did not report 
participating in any other pace-defined or modality-defined preparation)’ and to no 
preparation activity at all. 


Product-defined test preparation included only test preparation courses, only books, 
only online or software, multiple types of products, and no products at all. Student 
responses that could not be categorized were identified as “other” types of products. 


See Appendix A for more detail. 


Students reported engaging in only self-paced (27%) or only instructor-led (21%) test 
preparation activities (Figure 1). A total of 42% of students reported engaging in only 
non-online test preparation, which was the predominant modality. Approximately 
one in five students did not participate in any test preparation activity, and 29% 
reported not using any products. If a product was used, it was most often one of 
multiple products (26%). 
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Figure 1. Percentage of Students Using Test Preparation Activities and Products 
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Note: Students who selected “practice tests only’ and “no test preparation” are the 
same students across all defined test preparation activities. 


Test Preparation by Race/Ethnicity 


We found that test preparation defined by pacing, modality, and products varies by 
race/ethnicity.® Based on our findings, Asian and Black students are more likely than 
White and Hispanic students to engage in some type of pace-defined test 
preparation activity rather than do nothing at all.2 More specifically, the likelihood of 
White and Hispanic students not using pace-defined activities (23% and 22% of the 
time, respectively) is higher than for Asian (13% of the time) and Black (15% of the 
time) students (Figure 2). Asian students are predicted to participate the most in self- 
paced test preparation activities (36% of the time) and practice tests (23% of the 
time). Black students are predicted to participate the most in self-paced activities 
(27% of the time), followed by instructor-led activities (25% of the time). Likewise, 
Asian and Black students are also more likely than their counterparts to participate in 
both instructor-led and self-paced test preparation activities. Black students’ 
participation in a variety of test preparation activities and, more specifically, in 
instructor-led learning is consistent with previous research (Buchmann et al., 2010; 
Park & Becks, 2015). 
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Hispanic and White students are similar in terms of test preparation: students from 
both racial groups are predicted to participate the most in only self-paced activities 
(26% of the time for Hispanic students and 28% of the time for White students), 
followed by no preparation. 


Figure 2. Adjusted Probability of Participating in Test Preparation for Pacing, Mode, and 
Product, by Race 
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Note: Students who identified as a race other than those listed in this figure were omitted due to small 
sample sizes. The “both” pacing-defined activity includes both self-paced and instructor-led activities. 
The “both” mode-defined activity includes both non-online and online test preparation. For products, 
“multiple” means two or more of the remaining product types listed. 
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Regardless of race, students are predicted to participate the most in non-online test 
preparation activities (between 41% and 44% of the time) compared to the other 
Modalities. For Hispanic and White students, non-online test preparation is followed 
by no test preparation (21% and 22% of the time, respectively), but for Black and 
Asian students, it is followed by using practice tests (22% of the time for Asian 
students and 18% of the time for Black students). As with pace-defined activities, 
Black and Asian students are more likely than their Hispanic and White counterparts 
to participate in some type of mode-defined test preparation rather than do nothing. 
(It is important to keep in mind that the same students are in both the pace- and 
mode-defined analyses of their test preparation.) Finally, Asian students are predicted 
to participate in both online and non-online test preparation modes 15% of the time, 
and Black students 13% of the time; both percentages are higher than those 
predicted for Hispanic and White students (10% of the time). 


At 30% and 31%, respectively, Hispanic and White students are most likely to not 
participate in any product-defined test preparation (e.g., books, software, courses). 
However, Black and Asian students are most likely to use multiple test preparation 
products (32% and 35%). 


Test Preparation by Family Income 


We also investigated whether test preparation pacing, mode, and product types vary 
by family income (low: less than $36,000 a year; moderate: between $36,000 and 
$100,000 a year; high: more than $100,000 a year)."° 


Compared to students from high-income families, students from low-income and 
moderate-income families are more likely to not participate in any test preparation at 
all: students from high-income families are predicted to participate in no test 
preparation 19% of the time, students from low-income families 21% of the time, and 
students from moderate-income families 23% of the time (Figure 3). In addition, 
students from high-income families participate in instructor-led test preparation 26% 
of the time, but this is lower for students who come from moderate- and low-income 
families (17% and 18% of the time, respectively). Conversely, low- and moderate- 
income students are more likely than high-income students to participate in only 
self-paced preparation or to only take practice tests. This might be because of the 
free or low-cost options that are afforded by these two test preparation strategies. 


In terms of income differences for modality-defined test preparation activities, 
students from high-income families participate in non-online modes 47% of the time. 
This is higher than for those students who come from moderate-income (39% of the 
time) and low-income (42% of the time) families. For surveyed students who engaged 
in no test preparation or only took practice tests, differences by family income in this 
modality-defined analysis are similar to those in the pace-defined analysis. There are 
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no substantial differences by family income for the types of products used to prepare 
for the ACT. 


Figure 3. Probability of Participating in Test Preparation Activities for Pacing, Mode, and 
Product, by Family Income 
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Test Preparation by Parents’ Education Levels 


We also investigated whether test preparation varied by parents’ education levels (no 
parent with a bachelor’s degree versus at least one parent with a bachelor’s degree or 
higher)." There were only a few noticeable differences, which we highlight here. 


Based on the survey results, students who have a parent with a bachelor’s degree or 
higher are predicted to prepare for the ACT using practice tests 22% of the time, 
compared to 17% of the time for students who do not have parents with degrees 
(Figure 4). Interestingly, for modality-defined test preparation, students with parents 
who have not earned degrees engage in non-online test preparation activities 45% of 
the time, whereas students who have a parent with a degree engage in this kind of 
preparation 40% of the time. This unexpected finding might be because we 
controlled for income when investigating parents’ education levels. In fact, this might 
indicate a more nuanced understanding of parental involvement with test 
preparation. It might be that instead of parents’ education levels (and in turn, their 
levels of involvement) having a direct or more obvious relationship with the mode of 
test prep, education levels and involvement interact with other factors (e.g., income 
or race) to have a more salient effect on test prep model (Devine-Eller, 2012). Another 
possible explanation is that there are just more non-online preparation options 
available to all students, and underserved students take advantage of these options. 
This might be especially true for those underserved students who are less likely to 
have access to technological devices and who live in homes with unreliable internet. 
In addition, students who have parents with no college degrees use multiple 
products to prepare for the ACT 27% of the time; this occurs 23% of the time for 
students who have a parent with a bachelor’s degree or higher. Finally, 
online/software products are used 16% of the time by students with a parent who has 
a bachelor’s degree or higher and 13% of the time by students who have parents with 
no college degrees. 


Pacing 


Modality 


Product 
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Figure 4. Probability of Participating in Test Preparation Activities for Pacing, Mode, and 
Product, by Parents’ Education Levels 
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Test Preparation Activities and Products in Relation to 
ACT Scores 


Given the variety of test preparation activities and products that students use, we 
wanted to investigate next which of these test preparation features (pace, mode, and 
product) might be related to students’ ACT Composite scores (Figure 5).'” 


Figure 5. Mean ACT Composite Score by Pacing-, Modality-, and Product-Defined Test 
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*Statistically significant at .0S when comparing the test preparation initiative to no test preparation. 
Note: Each test preparation activity is represented by a box. The middle line represents the adjusted 
mean ACT Composite score. The top portion of the box represents the upper confidence interval, and 
the bottom portion of the box represents the lower confidence interval around that adjusted mean. 


On average, students who participate only in activities that are instructor-led are 
expected to earn an ACT Composite score of about 23.0, and those who participate in 
both self-paced and instructor-led preparation are expected to earn an ACT 
Composite score of 22.9, both of which are higher than the average scores for those 
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students who do not participate in test prep at all (22.2) and those who participate in 
self-paced test preparation only (22.3). 


With regard to the mode of test preparation, a student who participates in only non- 
online test preparation activities is expected to earn an ACT Composite score, on 
average, of about 22.8, which is higher than scores earned after doing no preparation 
at all (22.2) and doing both online and non-online preparation (22.3). 


For test prep products, a counterintuitive result emerged from the survey. Students 
who did not use any preparation products or who participated in preparation courses 
earned, on average, higher ACT Composite scores (22.9 and 22.7, respectively) than 
those who used other test preparation products—online or software only (22.2), other 
products (22.3), books only (22.4), or multiple products (22.4). It is unclear why this is 
the case. One possible explanation is that while the survey item about pace- and 
mode-defined test preparation focused on engagement in activities (e.g., self-paced 
review of content), the items about products focused mostly on whether students 
had access to each product, with no emphasis on how well—or even if—the products 
were used. 


Discussion 


The type of ACT preparation students engage in varies by racial/ethnic group and, toa 
lesser extent, by the students’ family income. Black students are likely to engage in 
some type of pace- and modality-defined test preparation, and they use multiple 
products. Asian students are less inclined to participate in instructor-led test 
preparation but more inclined to participate in self-paced activities and use multiple 
products. Hispanic students’ patterns of test preparation are similar to those of White 
students. Students from high-income families are more inclined to participate in 
instructor-led test preparation than students from low- and moderate-income 
families. Similarly, high-income students are more inclined to participate in test 
preparation that is not conducted online. 


It is important to highlight the benefits that instructor-led test preparation is 
expected to have on students’ ACT Composite scores, since it alone or in combination 
with self-paced test preparation was related to higher test scores among our survey 
participants. It is not surprising that students who come from families with lower 
incomes tend to be less likely to participate in this type of test preparation. We 
therefore recommend that students from more economically disadvantaged families 
be provided with opportunities to engage in instructor-led test preparation. This 
might include a scholarship program for individual students who would not 
otherwise be able to afford this experience or an in-school test preparation program 
where students from low-income families could gain exposure to instructional 
services. Opportunities like these could also be targeted to Hispanic, White, and Asian 
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students, since they too participate in instructor-led events less frequently than their 
Black counterparts. 


We also found that out of all types of mode-defined test preparation, non-online test 
preparation had the greatest relationship to students’ ACT Composite scores. While it 
was nice to see that often-marginalized groups like students of color and students 
whose parents do not have degrees were more inclined to participate in this type of 
test preparation, we also noted that this type of preparation was less frequently used 
by students from low- and moderate-income families. We therefore also recommend 
that all students receive access to test preparation that moves beyond online 
learning. Given the appeal and cost-effectiveness of online learning, we recommend 
that the online format increase its potential impact on learning by focusing on 
individualized learning components tailored to the needs of each student. We also 
emphasize the importance of keeping students engaged and motivated, and of 
providing them with clear instructions on how to use the online material (Bransford, 
Brown, & Cocking, 1999; Chen & Jang, 2010; Dede, 2008). 


We conclude with a few observations. First, though we have predicted which types of 
pace-, mode-, and product-defined test preparation are related to students’ ACT 
Composite scores, we acknowledge that that trend could, in fact, go in the opposite 
direction. That is, students with higher ACT scores might gravitate toward certain 
types of test preparation strategies. In this sense, then, ACT scores (i.e., achievement) 
might predict test preparation approaches.” Second, our research has focused on the 
test preparation that students report engaging in. We still know very little about their 
level of access to test preparation resources (e.g., family support for test preparation) 
and where these resources might be available (e.g., at school). We were also not able 
to investigate how much effort students put in to engaging with the test preparation 
materials or how each type of test preparation was implemented. We recommend 
that future research investigate this further.” 
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Appendix A 


Table A.1. Test Preparation Activities Aligned to Pace-Defined and Mode-Defined Groups 


AVeMYYColUI Ce Malena C-MxoM- lt GYolem-]ofolUi male VMtolUM ola-) or-1e-co Mill 
at all, for the February 6th ACT test. Which of the following 


i<-X- ow ola :) oy-Te-La (ola b- Loan hia (Mi mle lVAre i fe AVoLUMURY-M coll ole-)o}-le-Mielg 
the February 6th ACT test? (Choose all that apply.) 


Self-paced review of the content on the ACT using paper materials Self-paced Non-online 
(e.g., workbooks) 

Self-paced online review of the content on the ACT (excluding Self-paced Online 
online videos) 

Self-paced online review of the content on the ACT, using online Self-paced Online 
videos 

In-person small group instruction (less than 6 students) Instructor-led Non-online 
In-person large group instruction (6 or more students) Instructor-led Non-online 
In-person 1-on-1 tutoring Instructor-led Non-online 
Online small group instruction (less than 6 students) Instructor-led Online 
Online large group instruction (6 or more students) Instructor-led Online 
Online live 1-on-1 tutoring Instructor-led Online 
Other (please explain) Instructor-led Non-online 
Only practice assessments (e.g., practice tests, drills, or quizzes )* Practice Practice 

| have not participated in any test preparation activities No prep No prep 


*Students who chose “only practice assessments” were unable to choose the other activities. 


Table A.2. Test Preparation Activities Aligned to Pace-Defined and Mode-Defined Groups 
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use in preparation for the February 6th ACT test? (Choose all 


that apply.) Product 

ACT Online Prep Online/software only 
The Real ACT Prep Guide Books only 
Full-length practice tests available on actstudent.org Online/software only 
Sample items available on actstudent.org Online/software only 
Question of the day available on actstudent.org Online/software only 
Kaplan Prep course only 
Princeton Review Prep course only 
Sylvan Prep course only 
Test preparation books not published by ACT Books only 

Test preparation software not developed by ACT Online/software only 
High school prep course Prep course only 
Revolution Prep Prep course only 
Other (please describe) Other only 


If multiple types were chosen Multiple 
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Notes 


™On national 2018-2019 ACT test dates, students were asked “Did you prepare for the 
ACT® test using any test preparation materials (for example, The Official ACT® Prep 
Guide, other study guides, online materials, practice tests, tutors, or test prep 
courses)?” This percentage is based on students who responded to the question. 


2 More specifically, Messick said that the overarching purpose of test preparation was 
improving test scores either “by improving the skills measured by the test or by 
improving the skills for taking the test, or both.” 


3 An online survey was administered to a stratified random sample of students (N = 
45,400) who had registered to take the ACT in February 2016. Students who self- 
reported as Asian, Hispanic, or African American were over-sampled. White students, 
those who chose two or more races, and those who did not provide a race/ethnicity 
were under-sampled. All available American Indian/Alaska Native and Native 
Hawaiian/Other Pacific Islander students were included. Students who had retested 
more than one time prior to the February test date were also over-sampled; first-time 
test takers were under-sampled. This sampling method was utilized to ensure 
enough representation across all racial/ethnic and test-taking groups. Weights were 
used to analyze the data. Weighting reduces the chances for non-response bias and 
corrects for the stratified sampling conducted, which, on key measures expected to 
relate to test preparation behavior, makes the survey respondents more 
representative of the February 2016 tested population. Propensity score weighting 
was employed. Here, a logistic regression model was estimated predicting the 
probability of survey participation given population characteristics. We used 
race/ethnicity, gender, ACT Composite score, high school GPA, the number of times a 
student took the ACT, parents’ education levels, the number of times a student 
tested, and parents’ income levels as predictors. Missing data were imputed prior to 
inferential analyses. 


“ Students had to have answered the questions regarding the types of test 
preparation activities, if any, they participated in for the February 2016 test. The 
response rate was approximately 15%. 


5 We believe our work adds substantial value to the field because, unlike previous 
research on test preparation, it measures students’ engagement in test preparation in 
terms of all possible combinations of test preparation activities (as opposed to forcing 
students into the “highest impact” test preparation), allowing us to see whether 
engagement in multiple activities makes a difference. 


© In this study, attributes of test preparation activities were derived from two survey 
questions: “Which of the following test preparation activities, if any, did you use to 
prepare for the February 6th ACT test? (Choose all that apply.)” and “Which of the 
following test preparation products, if any, did you use in preparation for the February 
6th ACT test? (Choose all that apply.).” For the first question, test preparation activities 
were classified in terms of pacing (who runs it and if it is on a set schedule)— 
completely self-paced, completely instructor-led, both self-paced and instructor-led, 
only practice tests but nothing else, and no test preparation at all. The activities from 
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this first question were also classified in terms of their mode of delivery—completely 
online, completely non-online, both online and non-online, practice tests only, and no 
preparation at all. The test preparation product attribute was derived from the 
second question—whether the student used test prep courses only, books only, 
online/software test prep only, multiple products, other types of products, and no 
products at all. 


7 Students who indicated that they prepared for the ACT using only practice tests 
were unable to choose other test preparation activities in the survey. Therefore, 
students who provided this response were separated in the analysis from other 
pacing and modality offerings. 


8 A total of 60.1% of the sample was White, 4.7% was Asian, 16.6% was Black, 13.1% was 
Hispanic or Latino, 5.0% belonged to another race/ethnicity, and 0.5% did not report 
their race/ethnicity. 


° We estimated a series of multinomial regression models to determine which test 
preparation activities (by pace, modality, and product) students participated in and 
whether that participation varied by race, family income, and parents’ education 
levels. In this modeling, White student responses were the reference category for 
participation by race. A graduate degree or higher was the reference category for 
parents’ education levels. High family income was the reference category for family 
income level. We tested the relationship between race/ethnicity (and, subsequently, 
income and parents’ education levels) and each of the test preparation attributes 
(pacing, mode, product), controlling for demographic, academic, and non-academic 
factors (i.e., one Multinomial regression model for each test prep attribute with 
multiply imputed data). The factors (aside from race/ethnicity, income, and parents’ 
education levels) used in the model to predict each of the test preparation attributes 
consisted of gender, the number of times the student took the ACT, whether the 
student’s school is required to administer the ACT, the student’s college aspirations 
(i.e., degree level), the metro classification of the student’s school, whether the 
student is in their senior year, whether the student took a college curriculum, high 
school GPA, whether the student identified an educational need, the number of AP 
courses taken, and reasons the student participated in test preparation. We then 
calculated the predicted probabilities of an event occurring by race, family income, 
and parents’ education. Predicted probabilities were calculated with the mean values 
for all other variables in the model. It is these adjusted values, controlled for all other 
variables in the model, that we present in this paper. 


1 18.5% of the student sample was from a low-income family, 29.4% was from a 
moderate-income family, and 24.3% was from a high-income family; 27.8% did not 
report their family income. 


"51.8% of the sample had a parent with a bachelor’s degree or higher, 33.3% had 
parents without bachelor’s degrees, and 14.9% did not report their parents’ education 
levels. 


2 In examining the effects of test preparation engagement on ACT test scores, we 
calculated the estimated marginal means of the ACT Composite score for each type 
of test preparation activity, adjusting for any other academic or demographic 
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variables. The estimated marginal means are based on a statistical model of the ACT 
Composite score using the test preparation attributes and student’s academic and 
demographic background. Since both the pacing type and delivery mode were 
derived from the same survey question, we created two separate score models, each 
of which used one of the attributes as the test prep activity predictor; test prep 
product was included in both models. We also tested the estimated marginal means 
of the ACT Composite score on the interaction between the different groups of 
students (i.e., groups defined by race/ethnicity, family income, and highest parental 
education) and the types of their test preparation activities, but this did not yield any 
significant differences in scores. 


3 While we did not include ACT Composite scores in the models used to predict pace- 
, mode-, and product-defined test preparation, we did use high school GPA, another 
indicator of academic achievement. GPA as an indicator of achievement was never a 
statistically significant predictor of the types of test preparation activities the 
students engaged in. 


“% The study has other limitations. For example, while we weighted responses to 
ensure that the students were representative of our population, we focused our 
population only on students who took the test in February 2016. Likewise, we were 
limited by the survey design approach, including the items we asked and the 
response options provided. 


